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ABSTRACT 


A basic approach to the problem of evaluating or predicting a crew's 
performance for the VP community is presented. The method uses an ap- 
plication of multiple regression analysis techniques to a model which 
has training parameters as its variables. 

The results would allow the squadron or wing commanding officer to 
predict a crew's performance before the actual flight and to determine 


how to allocate training time for the squadron. 
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I,  IJNDRODUCTION 


A. BACKGROUND 

The patrol squadron ASW (VP) community has been trying to determine 
a method of evaluating crew performance effectively for many years. They 
not only would like to evaluate a performance by a crew but also to pre- 
dict a level of performance by a crew before the flight. The advantages 
of having such a system are obvious. A continuing performance chart of 
each crew, in each squadron would be available. With such a device 
WINGSPAC might be able to do away with yearly Operational Readiness In- 
spections (ORI). These evaluations have been used to check the ability 
of the squadron to meet any possible contingency. It was given through 
a multitude of flight, written examinations, and ground evolutions. 
Basically it tested the squadrons training program and determined if a 
squadron was ready for deployment. By having these up to date performance 
charts of all crews the Wing could tell at any time, not just at a 
scheduled one (as with the ORI), if that squadron was up to par. 

On this same line the Wing commander could see from these charts if 
a certain squadron's overall perfornance trend was decreasing. If this 
was accompanied by a fee period of operational tasking it could mean 
the squadron should be stood down from operational flying so it could 
devote more time to training. The problem of over erekine a squadron 
with operational commitments is a serious problem today facing the Wing 
commander, 

In addition the squadron or Wing commander would be able to determine 
before the flight if a particular crew is qualified to go on that mission 


and do a job of a required degree of skill and competence. 





B. NATURE OF THE PROBLEM 

At the present time squadrons in WINGSPAC are using a postflight 
evaluation method in an attempt to evaluate a crew's performance. The 
crew is evaluated by a team of debriefers from the various ASCAC'S upon 
completion of the flight. The form is quite lengthy and requires con- 
siderable time to complete during the debrief and after. The range of 
grades goes from 0 to 100 for each area. The number of required areas 
to be graded being dependent on the type of flight flown. 

The major problem here is the standardization of the evaluators in 
how they grade. The variance in their attempts for objectivity in 
quantifying quality has been extremely high. Another area for concern 
is the length of the form. This combined with the large number of 
flights graded and relatively small number of evaluators available causes 
a rushed evaluation with not enough emphasis on constructive criticism. 
Last but not least is the fact that the evaluators are not completely 
devoid of personal connections to the people they are evaluating. 

Because of the above problems the average grade of a sample of 
flights of crews from several squadrons at Moffett Field is the un- 
believably high 96-97% and is attacked by WINGSPAC and the Tactical 
Support Center (TSC) at Moffett Field now. 

The problem that is attacked here is that of predicting a crew's 
performance. This problem is difficult because of the large number of 
variables affecting performance and the lac of, in some cases, of a 
suitable measure of that variable. 

Because of this it was decided to start by looking strictly at the 
major variables associated with training. The amount and type of train- 
ing might vary between squadrons but it is recognized as being a major 


mecessity tor all VP crews. 





A basic approach using multiple repression analysis was then under- 
taken with the selected training variable as independent parameters and 
the postflight evaluation score as the dependent variable. 

The inability of the squadrons to collect data in the required 
format did not allow us to use real world data to test proposed models. 
Data was generated by the author, however, to demonstrate the techniques 
involved. The data used is a reasonable attempt to simulate the actual 


data, based on prior experience. 
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A. THE MODEL 


The independent variables chosen to be looked at are: 


Xo Total flight time - the time in hours 
flown as a crew since the crew was formed, 
Xy Weapons Systems Trainer (WST) time - time 
in hours in trainer as a crew. 
xy, : Sub time - time in hours of actual on station 


time working a submarine as a crew. 


The dependent variable is: 
xy : Postflight evaluation score - overall 
grade of evaluators given as a percent. 
It is expected that the value of X, would increase with an increase 
in the independent variable. The rate of increase should decrease as 


the independent variables get larger. One method of mathematically 


describing such a relationship is given by: 


KX, = l- ees ee (1) 
where 
Osx Xi =I 
Os opie 
Xo » Xe 5 xy, =o 18) 
Let Z = AX, + AX, = a, X),- In equation (1) as the value of Z 


approaches infinity then the value of Xy approaches 1 because the value 
of ec = 0. Similarly as the value of Z approaches O then the value 


of Xy approaches B because ao = ih 
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Figure (1) 
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be cAPPELCALION 

Before the model could be used in a linear regression analysis it 
had to be transformed from its non-linear form in equation (1) into a 
linear form. This can be done by using a logarithmic transformation on 


equation (1). Regrouping terms equation (1) becomes 


a7 A yXo + a4X, + a, X,) (2) 


Now applying the transformation by taking the natural logarithm of 


both sides equation (2) becomes 
y = In (1 - X,) = np - WyX, ~ aX, ~ aX, (3) 


The natural base for the logarithm was chosen for convenience only. 
In this form it is ready to be used in conjuntion with any available 
computer multiple regression package, The author has used the BIMEDO2R 
stepwise linear regression package [| Ref. 1] on the IBM 360 computer. 
This program allows variables to enter one at a time and permits the 
user to see in what order and with what effect each does. 
A 


For manual calculations let b be a vector containing the estimators 


of the constant term inf, Was Aas and ay et7 ine 


o> 
i] 
R> R> > 
NO 


Q> 
om 


where the symbol “* indicates that a quantity is an estimator. 





Let y be a matrix of the observations on all independent variables 


augmented by a column of units representing the constant term. 


[be Os X X 
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and yx' is y transpose 
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Let y be a vector of observations of the dependent variable. 


The least squares estimates are given in matrix notation by 
: 1 al 1 
De = 1G 0 Saas So) 


If the problem was reduced to only one independent variable, equation 


(3) would simply be 


is (l= 3G) Sa Ms] ae 
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or by some basic substitutions 


yy = 2A ee. 


where A = I1nB and B = - e, 


Solving for the estimates of A and B by linear regression the re- 


sults are: 


> 
> 


Cee Y) 


i=l 
ae 2 
yO. Cer O 
; i 
i=l 
marere the — denotes the mean value of all observations of that variable. 
A point estimate of y given a particular value of X can be 


found from the equation 


“A Ae 


y= A + BX 


where Y is the point estimate and iced particular value of X. 
: a 


To use the t-test we must solve for the standard deviation of B. 


This is given by 





Bas 


La 
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C. DATA COLLECTION 

Data has to be collected by the fleet to test the model. It could 
be done in the following manner. For any flight that is to be graded 
the tactical coordinator of the crew can give the values of the variables 


Xo» Xy , X, either during the brief or the debrief. After the evaluators 


Z 


have finished grading the flight they will record the overall score as 


variable Xy and turn into collection point along with squadron, crew, 


event number, and date. These values of X Xo » Xe and X, form one 


2 


set of observations. When a reasonable number have been gathered they 


ls 


can be used with the regression package and run on the computer or using 
formulas (4) and a calculator the model can be tested and the estimators 
found. Once the estimator of the constant In8 and the estimators of the 
coefficients Wns Wns and aw, have been determined these values can be 


i 


used to predict future values of the dependent variable X)- 
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TII. DATA ANALYSIS 


Summary of computer results using data in Table 1 


coefficient standard: error 
X -.00105 00076 
Xy - .00698 s0027 2 
Xy, BOON) 2 00994 
constant Boi] 
multiple R soo09 
R square 1/848 


aA 
A t-test is used to test the hypothesis that the value of O. is 


equal to ¢. The hypothesis is accepted if 


a 
oe 
et BS ee Be ape ey 
Coon 1 or Cpe 1 
i 
ras 
and rejected if i lies outside this interval. In the avove equation 
oo 
Y. 
of 


c is equal to 1l- confidence level desired and n-l is called the degrees 
of freedom, where is the number of observations. 
A t-table can be found in [Refs. 2,3,4, and 5] and many other 


textbooks. Part of a typical table is shown below: 


: L670 2.009 





One would enter with the degrees of freedom (df) equal to n-1 down 


the left column and the value of c/2 across the top. 
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For a 90% confidence level for the given data we would enter with 


df€ = 39 and ¢c/2 = .05. The value obtamiea was). cos 


-00105 
- 1,684 < - 00076 < 1.684 
= Nip OO Fie mien 3 Bill <. 6s 


This tells the user that the coefficient is not significant at the 
90% confidence level or that it is equal to ¢ at this level. It can be 
seen from the t-tables that it becomes significant at the 80% level. The 
t value being 1.303. 


Similarly w, can be tested. Using a 90% confidence level 


3 
.00698 
- 1.684 < “90272 1.684 
-1,.684 < - 2,564 < 1.684 
Here a’. 1S Slenusucant and ditrerent from 0. 
Just by looking at the computer results for aw, it can be seen that 


4 


the standard error is considerably larger than the coefficient itself. 

A 
This indicates that a, is not significant at any practical level of con- 
fidence. 

Another important figure in the results is the R square. This in- 
dicates that 78% of the variance in the data is accounted for by the 
model and its significant variables. 

Since a logarithmic transformation was used on the model any term 
in the final results so affected has to be transformed back to its 
usable form in the model. Here only the constant term is affected. 


Taking the antilog of -0.97327 with a natural base the resulting value 


ms . 35/779. 


14 





The equation for predicting future scores would be 


-(.00105X, + .00698X,) 


Xx 2 bee oe 2 


i 


With the data used the values of On and oe indicate the relative 
weighing factors of the training parameters Xn and Xn. Since Oo = AEWA IOS, 


“a 
and a, = .00698, it would indicate that the training denoted by X., would 


have approximately 6.5 times the effect on performance as the training 


denoted by Xo Again this is strictly for the data used in this analysis. 
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IV. CONCLUSIONS 


The approach to the problem is only a basic feasible starting point. 
The model presented may be one of many possible ones. The fleet has to 
take the model presented and test it with actual data collected on a 
higher authority basis, using the methodology presented here. The num- 
erical results found here do not apply to the fleet because they were 
arrived at using generated data. The implications from the tests show 
that the model could work and should be tested. Once this is done the 
estimators that are found can be used to predict the performance of any 
crew on a pending flight. 

An important secondary use of these coefficient estimators is as 
a weighting factor to determine allocation of training time within a 
squadron. The coefficients form a relative merit scale of each parameter 
used in the regression. 

The user can add as many independent variables as he chooses with- 
out complicating the methodology. Since the BIMEDO2R is a stepwise 
regression model, the user is able to determine the effect of these 


8 


variables individually if he can collect the data in a usable form. 
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